3D OPIN-CHANZ connectivity #3323

soheilshahrouz · 2025-10-28T15:53:20Z

Removes the pin_offset attribute from the architecture file, which was previously used to model 3D OPIN connectivity.
Instead, the <fc_override> tag is now used to specify absolute fc values for connecting OPINs to CHANZ wire segments.

Additionally, several functions have been moved from rr_graph2.cpp to two new files: rr_graph_chan_chan_edges.cpp and rr_graph_opin_chan_edges.cpp.

…raph_sg

AmirhosseinPoolad

What a PR, Soheil. Had some thoughts, mostly on styling. I did not review rr_graph_chan_chan_edges.cpp/h and rr_graph_opin_chan_edges.cpp/h as you said it was only moving the code to a new file.

Also only reviewed add_edges_opin_chanz_per_side and add_edges_opin_chanz_per_block in rr_graph_sg for the same reason.

AmirhosseinPoolad · 2025-11-12T20:10:52Z

libs/librrgraph/src/base/get_parallel_segs.h

                                             e_parallel_axis parallel_axis,
                                             bool keep_original_index = false);
+
+int get_parallel_seg_index(int abs,


Needs doxygen

AmirhosseinPoolad · 2025-11-12T20:18:10Z

libs/librrgraph/src/base/rr_spatial_lookup.h

                                                       e_rr_type rr_type) const;

+
+    std::vector<RRNodeId> find_pin_nodes_at_side(int layer,


AmirhosseinPoolad · 2025-11-12T20:25:28Z

vpr/src/route/rr_graph_generation/build_scatter_gathers.cpp

                                  scatter_wire_candidates.size());

-                    continue;
+                    // continue;


Remove commented out code

AmirhosseinPoolad · 2025-11-12T20:27:32Z

vpr/src/route/rr_graph_generation/clock_network_builders.cpp

+    seg_idx = get_parallel_seg_index(drive_seg_idx, indices_map, e_parallel_axis::X_AXIS);
+    drive_seg_idx = (seg_idx >= 0) ? seg_idx : drive_seg_idx;

-    get_parallel_seg_index(left_seg_idx, e_parallel_axis::X_AXIS, indices_map);
+    seg_idx = get_parallel_seg_index(left_seg_idx, indices_map, e_parallel_axis::X_AXIS);
+    left_seg_idx = (seg_idx >= 0) ? seg_idx : left_seg_idx;

-    get_parallel_seg_index(right_seg_idx, e_parallel_axis::X_AXIS, indices_map);
+    seg_idx = get_parallel_seg_index(right_seg_idx, indices_map, e_parallel_axis::X_AXIS);
+    right_seg_idx = (seg_idx >= 0) ? seg_idx : right_seg_idx;


Rename drive_seg_idx, left_seg_idx and right_seg_idx to have underlines after them. Not clear in a glance that they're private fields and not consistent with the rest of the code base.

Explain the logic of this code. Not sure why we are doing the set either the reutnred value or the old value as the new value and that looks like new behaviour. It should be explained.

AmirhosseinPoolad · 2025-11-12T20:29:10Z

vpr/src/route/rr_graph_generation/rr_graph.cpp

 *   @param perturb_switch_pattern Specifies whether connections should be distributed unevenly across the channel or not.
 *   @param directionality Segment directionality: should be either *UNI-DIRECTIONAL* or *BI-DIRECTIONAL* 
 *   @param seg_inf Segments type information, such as length, frequency, and etc.
 *   @param sets_per_seg_type Number of available sets within the channel_width of each segment type.
 * 
- * @return an 5D matrix which keeps the track indicies connected to each pin ([0..num_pins-1][0..width-1][0..height-1][0..layer-1][0..sides-1]).
+ * @return an 4D matrix which keeps the track indices connected to each pin ([0..num_pins-1][0..width-1][0..height-1][0..layer-1][0..sides-1]).


It's still 5 dimensions in the comment. '([0..num_pins-1][0..width-1][0..height-1][0..layer-1][0..sides-1])'

layer should be removed from the comment I think.

AmirhosseinPoolad · 2025-11-12T22:36:31Z

vpr/src/route/rr_graph_generation/rr_graph_sg.h

+                                   t_rr_edge_info_set& rr_edges_to_create,
+                                   const vtr::NdMatrix<std::vector<t_bottleneck_link>, 2>& interdie_3d_links);
+
+void add_edges_opin_chanz_per_block(const RRGraphView& rr_graph,


AmirhosseinPoolad · 2025-11-12T22:37:47Z

vpr/src/route/rr_graph_generation/rr_types.h

 * [0..device_ctx.physical_tile_types.size()-1][0..num_pins-1][0..width][0..height][0..layer-1][0..3][0..Fc-1] */
-typedef std::vector<vtr::NdMatrix<std::vector<int>, 5>> t_pin_to_track_lookup;
+typedef std::vector<vtr::NdMatrix<std::vector<int>, 4>> t_pin_to_track_lookup;


Update the comment above (remove layer)

AmirhosseinPoolad · 2025-11-12T22:39:19Z

vpr/src/route/rr_graph_generation/rr_types.h

 * 
 * Note that when we model different channels based on position not axis, we can't use this anymore and need to have a lookup for each grid location. */
-typedef std::vector<vtr::NdMatrix<std::vector<int>, 5>> t_track_to_pin_lookup;
+typedef std::vector<vtr::NdMatrix<std::vector<int>, 4>> t_track_to_pin_lookup;


Update comment above.

AmirhosseinPoolad · 2025-11-12T22:40:56Z

vtr_flow/arch/multi_die/simple_arch/3d_k4_N4_90nm.xml

+        <!-- Gather 30 connections from the end of L4 wires at all four sides of a switchblock location -->
+        <wireconn num_conns="0" from_type="L1" from_switchpoint="1" side="rltb"/>
+      </gather>
+      <scatter>
+        <!-- Scatter 30 connections to the start of L4 wires at all four sides of a switchblock location -->
+        <wireconn num_conns="30" to_type="L1" to_switchpoint="0" side="rtlb"/>
+      </scatter>
+      <sg_link_list>
+        <!-- Link going up one layer, using the 'seg4_inter_die_driver' multiplexer to gather connections from the bottom layer and using the LZ node/wire to move up one layer -->
+        <sg_link name="L_UP" z_offset="1" x_offset="0" y_offset="0" mux="segz_driver" seg_type="LZ"/>
+        <!-- Link going down one layer, using the 'seg4_inter_die_driver' multiplexer to gather connections from the top layer and using the LZ node/wire to down up one layer -->
+        <sg_link name="L_DOWN" z_offset="-1" mux="segz_driver" seg_type="LZ"/>
+      </sg_link_list>
+      <!-- Instantiate 6 'L_UP' and 6 'L_DOWN' sg_links per switchblock location everywhere on the device -->


Update the comments here to represent the architecture.

AmirhosseinPoolad · 2025-11-12T22:42:19Z

vtr_flow/arch/multi_die/stratixiv_3d/3d_full_OPIN_inter_die_stratixiv_arch.timing.xml

+  <!-- Specify a scatter-gather pattern for 3D connections. -->
+  <scatter_gather_list>
+    <sg_pattern name="name" type="unidir">
+      <gather>
+        <!-- Gather 30 connections from the end of L4 wires at all four sides of a switchblock location -->
+        <wireconn num_conns="0" from_type="L4" from_switchpoint="3" side="rltb"/>
+      </gather>
+      <scatter>
+        <!-- Scatter 30 connections to the start of L4 wires at all four sides of a switchblock location -->
+        <wireconn num_conns="30" to_type="L4" to_switchpoint="0" side="rtl"/>
+      </scatter>
+      <sg_link_list>
+        <!-- Link going up one layer, using the 'seg4_inter_die_driver' multiplexer to gather connections from the bottom layer and using the LZ node/wire to move up one layer -->
+        <sg_link name="L_UP" z_offset="1" x_offset="0" y_offset="0" mux="seg4_inter_die_driver" seg_type="LZ"/>
+        <!-- Link going down one layer, using the 'seg4_inter_die_driver' multiplexer to gather connections from the top layer and using the LZ node/wire to down up one layer -->
+        <sg_link name="L_DOWN" z_offset="-1" mux="seg4_inter_die_driver" seg_type="LZ"/>
+      </sg_link_list>
+      <!-- Instantiate 6 'L_UP' and 6 'L_DOWN' sg_links per switchblock location everywhere on the device -->
+      <sg_location type="EVERYWHERE" num="60" sg_link_name="L_UP"/>
+      <sg_location type="EVERYWHERE" num="60" sg_link_name="L_DOWN"/>
+    </sg_pattern>
+  </scatter_gather_list>


Update comments.

Make sure there is a comment at the top of the arch file explaining the 3D part.

vaughnbetz

Nice cleanup and overall well structured code. Needs more commenting though in places.

vaughnbetz · 2025-11-14T22:17:51Z

libs/librrgraph/src/base/get_parallel_segs.h

                                             e_parallel_axis parallel_axis,
                                             bool keep_original_index = false);
+
+int get_parallel_seg_index(int abs,


vaughnbetz · 2025-11-14T22:22:16Z

vpr/src/base/setup_vpr.cpp

 }

 /**
 * @brief This loads up VPR's arch_switch_inf data by combining the switches


I think we should move this doxygen comment to the declaration of the function near the top of the file.

vaughnbetz · 2025-11-14T22:35:14Z

vpr/src/route/router_lookahead/router_lookahead_map_utils.cpp

        int curr_layer_num = rr_graph.node_layer_low(curr.node);
-        if (curr_rr_type == e_rr_type::CHANX || curr_rr_type == e_rr_type::CHANY || curr_rr_type == e_rr_type::SINK) {
-            //We stop expansion at any CHANX/CHANY/SINK
+        if (curr_rr_type == e_rr_type::CHANX || curr_rr_type == e_rr_type::CHANY || curr_rr_type == e_rr_type::CHANZ || curr_rr_type == e_rr_type::SINK) {


Might want to add CHANZ in some of the ASCII art above this line.

vaughnbetz · 2025-11-14T22:35:43Z

vpr/src/route/router_lookahead/router_lookahead_map_utils.cpp

            }

+            if (curr_rr_type == e_rr_type::CHANZ) {
+                curr_layer_num = (root_layer_num + 1) % 2;


Explain why ... I'm not sure why this is done at all.

vaughnbetz · 2025-11-14T22:43:04Z

vpr/src/route/rr_graph_generation/clock_network_builders.cpp

+    seg_idx = get_parallel_seg_index(drive_seg_idx, indices_map, e_parallel_axis::X_AXIS);
+    drive_seg_idx = (seg_idx >= 0) ? seg_idx : drive_seg_idx;

-    get_parallel_seg_index(left_seg_idx, e_parallel_axis::X_AXIS, indices_map);
+    seg_idx = get_parallel_seg_index(left_seg_idx, indices_map, e_parallel_axis::X_AXIS);
+    left_seg_idx = (seg_idx >= 0) ? seg_idx : left_seg_idx;

-    get_parallel_seg_index(right_seg_idx, e_parallel_axis::X_AXIS, indices_map);
+    seg_idx = get_parallel_seg_index(right_seg_idx, indices_map, e_parallel_axis::X_AXIS);
+    right_seg_idx = (seg_idx >= 0) ? seg_idx : right_seg_idx;


Explain the logic of this code. Not sure why we are doing the set either the reutnred value or the old value as the new value and that looks like new behaviour. It should be explained.

vaughnbetz · 2025-11-15T00:01:19Z

vpr/src/route/rr_graph_generation/rr_graph_sg.cpp

+    const int grid_height = grid.height();
+
+    sb_loc0.x = std::clamp(sb_loc0.x, 0, grid_width - 1);
+    sb_loc0.y = std::clamp(sb_loc0.y, 0, grid_height - 1);


You may be able to shorten this routine by putting the sb_loc0 etc. into a vector with two elements and looping. Or you could use helper functions as @AmirhosseinPoolad suggests.

vaughnbetz · 2025-11-15T00:06:15Z

vtr_flow/arch/multi_die/simple_arch/3d_k4_N4_90nm.xml

+    <sg_pattern name="name" type="unidir">
+      <gather>
+        <!-- Gather 30 connections from the end of L4 wires at all four sides of a switchblock location -->
+        <wireconn num_conns="0" from_type="L1" from_switchpoint="1" side="rltb"/>


Should num_conns be 30 instead of 0 here?

vaughnbetz · 2025-11-15T00:06:54Z

vtr_flow/arch/multi_die/simple_arch/3d_k4_N4_90nm.xml

-          <loc side="top" layer_offset="1">io.outpad io.inpad io.clock</loc>
-          <loc side="right" layer_offset="1">io.outpad io.inpad io.clock</loc>
-          <loc side="bottom" layer_offset="1">io.outpad io.inpad io.clock</loc>
+          <loc side="left">io.outpad io.inpad io.clock</loc>


Add a comment at the top of this arch file explaining what it is.

vaughnbetz · 2025-11-15T00:08:12Z

vtr_flow/arch/multi_die/stratixiv_3d/3d_full_OPIN_inter_die_stratixiv_arch.timing.xml

+    <sg_pattern name="name" type="unidir">
+      <gather>
+        <!-- Gather 30 connections from the end of L4 wires at all four sides of a switchblock location -->
+        <wireconn num_conns="0" from_type="L4" from_switchpoint="3" side="rltb"/>


Should this be num_conns = 30?

vaughnbetz · 2025-11-15T00:08:35Z

vtr_flow/arch/multi_die/stratixiv_3d/3d_full_OPIN_inter_die_stratixiv_arch.timing.xml

+  <!-- Specify a scatter-gather pattern for 3D connections. -->
+  <scatter_gather_list>
+    <sg_pattern name="name" type="unidir">
+      <gather>
+        <!-- Gather 30 connections from the end of L4 wires at all four sides of a switchblock location -->
+        <wireconn num_conns="0" from_type="L4" from_switchpoint="3" side="rltb"/>
+      </gather>
+      <scatter>
+        <!-- Scatter 30 connections to the start of L4 wires at all four sides of a switchblock location -->
+        <wireconn num_conns="30" to_type="L4" to_switchpoint="0" side="rtl"/>
+      </scatter>
+      <sg_link_list>
+        <!-- Link going up one layer, using the 'seg4_inter_die_driver' multiplexer to gather connections from the bottom layer and using the LZ node/wire to move up one layer -->
+        <sg_link name="L_UP" z_offset="1" x_offset="0" y_offset="0" mux="seg4_inter_die_driver" seg_type="LZ"/>
+        <!-- Link going down one layer, using the 'seg4_inter_die_driver' multiplexer to gather connections from the top layer and using the LZ node/wire to down up one layer -->
+        <sg_link name="L_DOWN" z_offset="-1" mux="seg4_inter_die_driver" seg_type="LZ"/>
+      </sg_link_list>
+      <!-- Instantiate 6 'L_UP' and 6 'L_DOWN' sg_links per switchblock location everywhere on the device -->
+      <sg_location type="EVERYWHERE" num="60" sg_link_name="L_UP"/>
+      <sg_location type="EVERYWHERE" num="60" sg_link_name="L_DOWN"/>
+    </sg_pattern>
+  </scatter_gather_list>


Make sure there is a comment at the top of the arch file explaining the 3D part.

amin1377

Looks good, Soheil! Thanks! I don’t have much to add to what Amir and Vaughn already mentioned. Just one question: have you explained fc_override and how it helps with OPIN connections to other layers in the document? Thanks!

amin1377 · 2025-11-14T20:36:07Z

libs/librrgraph/src/base/rr_graph_utils.cpp

-                    limited_to_opin = false;
-                    break;
-                }
+            e_rr_type to_type = rr_graph.node_type(to_node);


It’s not directly related to this section, but it occurred to me while reading this part. I think we assume that CHANZ is definitely stretched across multiple layers. If that’s correct, do we have a check for this in the RR graph validation to ensure that assumption holds?

soheilshahrouz added 5 commits October 10, 2025 15:13

clean some comments

c3ea108

clean get_track_to_pins a bit

cd680d7

remove direct pin to other layer's track connectivity

8294d2d

remove arch_inter_die_switch

6857133

Merge remote-tracking branch 'origin/master' into temp_opin_chanz

271ed58

github-actions bot added VPR VPR FPGA Placement & Routing Tool libarchfpga Library for handling FPGA Architecture descriptions lang-cpp C/C++ code labels Oct 28, 2025

soheilshahrouz added 9 commits October 28, 2025 15:53

remove pin layer offset and ipin_cblock_switch_index_between_dice

bd51bd9

remove layer_offset from rr_graph.cpp

2c37b80

Merge remote-tracking branch 'origin/master' into temp_opin_chanz

5e5b118

doxygen comment for process_fc()

c5d509c

Merge branch 'temp_arch_xml_naming_convention' into temp_opin_chanz

a9b490a

doxygen comment for label_wire_muxes()

1f13be8

Fc for OPIN-CHANZ can be 1

4104e23

update the 3d_full_opin arch file

027ab8b

add OPIN--CHANZ edges

9210a3d

github-actions bot added the libvtrutil label Nov 1, 2025

soheilshahrouz added 12 commits November 1, 2025 15:03

move get_parallel_seg_index() to get_parallel_sets.cpp

f747061

move add_edges_opin_chanz to rr_graph_3d

230fead

remove pins_connected() function and rename rr_graph_3d files to rr_g…

0097c56

…raph_sg

move add_and_connect_non_3d_sg_links() to rr_graph_sg

d2af592

noexcept for strongid methods

21dfd5d

use the correct layer for CHANZ nodes when initalizing src_opin_delays

6803792

allow OPIN to be connected to CHANZ in check_adjacent()

1ede4c4

make format

ea14ba1

add find_pin_nodes_at_side() to RRSpatialLookup

78afdc2

add add_edges_opin_chanz_per_side()

0dcb2fa

update inter_layer_connections_limited_to_opin() to check OPIN-CHANZ

72e130c

some cleanups in rr_graph files

3610503

soheilshahrouz added 11 commits November 5, 2025 14:04

add rr_graph_opin_chan_edges files

a577480

doxygen comment for add_opin_chan_edges()

109896e

make format

d7b67ea

doxygen comments for some function in rr_graph_opin_chan_edges

9a12593

add rr_graph_chan_chan_edges

fd6d5c5

add rr_chanz_segment_width to DeviceContext

63cbe06

calculate acc_tile_num_inter_die_conn_ using pre-computed chanz width

5d477c1

Merge remote-tracking branch 'origin/master' into temp_opin_chanz

27cbc2a

update expected_delay_cost before VTR_ASSERT

eadc8fe

add sg and z segment to 3d_k4_n4_90nm

2657db0

remove duplication get_parallel_seg_index() function

55db1c8

soheilshahrouz changed the title ~~[WIP] 3D OPIN-CHANZ connectivity~~ 3D OPIN-CHANZ connectivity Nov 10, 2025

soheilshahrouz requested review from AmirhosseinPoolad, amin1377 and vaughnbetz November 10, 2025 23:00

soheilshahrouz added 2 commits November 10, 2025 18:09

update golden results for vtr_strong/strong_ap/basic_3d_ap/

644fe8a

don't update segment index if not found

e76138f

AmirhosseinPoolad reviewed Nov 12, 2025

View reviewed changes

vaughnbetz requested changes Nov 15, 2025

View reviewed changes

amin1377 requested changes Nov 15, 2025

View reviewed changes

		e_rr_type rr_type) const;


		std::vector<RRNodeId> find_pin_nodes_at_side(int layer,

3D OPIN-CHANZ connectivity #3323

Are you sure you want to change the base?

3D OPIN-CHANZ connectivity #3323

Uh oh!

Conversation

soheilshahrouz commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AmirhosseinPoolad left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vaughnbetz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amin1377 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

soheilshahrouz commented Oct 28, 2025 •

edited

Loading